I have not look into the code yet, but I've tried to run it locally and it looks like it's trying to load a lot of unnecesarry debug files.

I run ebpf profiler with no ontarget symbolization , also run a simple python -m http.server to mock debug infod responses.

I then query only one executable process_cpu:cpu:nanoseconds:cpu:nanoseconds{service_name="unknown", process_executable_path="/home/korniltsev/.cache/JetBrains/IntelliJIdea2024.2/tmp/GoLand/___go_build_go_opentelemetry_io_ebpf_profiler"}

I see 268 GET requests, with 13 requests to "GET /buildid/fbce2598b34f1cf8d0c899f34c2218864e1da6d1/debuginfo HTTP/1.1" 200 - (which is the profiler binary I put into mock server for testing and a bunch of 404 which I assume are build ids for the filles in the other processes which the query does not target.

Other then that it works \M/ Can't wait to run it in dev.

Hi @marcsanmi , when can this PR be merged?
Thanks

Hi @liaol,
It's still going to take a little while :)

I've created this diagram to outline the current Symbolization arch:

flowchart TD
    A[SymbolizePprof] --> B{Group by Mapping}
    B --> C[Symbolize Request]
    
    C --> D{Check Symbol Cache}
    
    subgraph "Symbol Cache Layer (LRU, in-memory)"
        D -->|Cache Hit| E[Return Cached Symbols]
        D -->|Cache Miss| F
    end
    
    F{Check Debug Info Cache} 
    
    subgraph "Debug Info Cache Layer (Ristretto, in-memory)"
        F -->|Cache Hit| G[Read from Debug Info Cache]
        F -->|Cache Miss| H
    end
    
    subgraph "Persistent Storage Layer"
        H{Check Object Store}
        H -->|Cache Hit| I[Read from Object Store]
        H -->|Cache Miss| J[Fetch from Debuginfod]
    end
    
    I --> K[Store in Debug Info Cache]
    J --> L[Store in Debug Info Cache]
    J --> M[Store in Object Store]
    
    G --> N[Parse ELF/DWARF]
    K --> N
    L --> N
    
    subgraph "DWARF Resolution Layer"
        N --> O[Resolve Addresses]
        O --> P{Check Address Map}
        P -->|Map Hit| Q[Return from Map]
        P -->|Map Miss| R[Parse DWARF Data]
        R --> S[Build Lookup Tables]
        S --> T[Store in Address Map]
        T --> U[Return Symbols]
        Q --> U
    end
    
    U --> V[Update Symbol Cache]
    V --> W[Return to Caller]
    E --> W

I might be missing some details, but I have doubts about the cache hierarchy.

Now it looks like we have: symbols_cache -> object_store -> in_memory_object_store (ristretto) -> debuginfod.

As far as I understand, we're going to read from object_store even if there's just a single unresolved address.

I expected to see: symbols_cache -> in_memory_object_store (ristretto) -> object_store -> debuginfod.

Could you please elaborate on the decision?

I might be missing some details, but I have doubts about the cache hierarchy.

You're right @kolesnikovae. I've just realized the problem is that the ristretto cache is coupled inside the debuginfod client. I'll decoupled it and placed it at symbolizer level. Thus, we'll be able to have the following path:

symbols_cache -> in_memory_object_store (ristretto) -> object_store -> debuginfod